1. Field of the Invention
The present invention relates to video data processing.
2. Background Art
An increasing number of devices are being produced that are enabled to capture and display images. For example, mobile devices, such as cell phones, are increasingly being equipped with cameras to capture images, including still snapshots and motion video images. Images captured by such devices can frequently be viewed on displays of the devices, as well as being transferred from the devices for viewing elsewhere. To view the images on relatively small devices, the images typically must be viewed on small display screens that are not capable of viewing the full resolution of the captured images. Thus, such devices must include at least limited image processing capability to down-size the images for viewing on the small display screens.
Many mobile devices have limited processing capability due to cost, power consumption, and size constraints. However, the processing of captured images, especially the processing of video, is very computationally intensive. For example, many mobile devices have cameras capable of capturing images of 2 MegaPixels (MPel) or more. Thus, a processor of such a mobile device must be capable of processing a large amount of data for each captured image. Furthermore, encoding and decoding (e.g., QCIF) of image data may need to be performed by the processor at frame rates such as 15 fps and 30 fps, respectively, as well as the performance of other functions.
Due to cost and portability concerns, a camera of a mobile device tends to have a small lens size, and the f-number of the lens (the ratio of lens focal length to lens diameter) is usually fixed or supports a very limited range of choices. To enable a sufficient amount of light to reach the camera sensor under various lighting conditions, a common strategy is to add an automatic exposure control (AEC) mechanism. AEC adaptively changes an exposure time, aperture size, signal gain settings, and/or other camera parameters according to detected lighting conditions.
A camera exposure time tends to be longer for dark/indoor settings and shorter for bright/outdoor settings. As a result, the video data output from the camera includes a higher number of frames per second for bright settings than for dark settings. Most camera module designs can have a wide range for the video frame rate. For example, variable frame rates from 5 fps to 30 fps, depending on the particular light conditions, are not uncommon. A video encoder that receives the video data must be able to adapt to the different frame rates. Whether or not the video encoder can adapt well is a key to high-quality video recording end-user experiences.
Practical video encoding applications are constrained either by a limited storage capacity of the recording media or the channel bandwidth of the communication link. Various video data rate control algorithms have been developed to deal with these limited resources. Such rate control algorithms, however, typically assume that video data is received at a constant frame rate. In some existing approaches, video frames that are received at a variable frame rate are re-sampled at fixed time instants to model a fixed frame rate source. Some other existing approaches estimate the frame rate based on the incoming video data, and then apply the estimated frame rate to future incoming data. Because video capture operations in small handheld devices are typically performed on a more instantaneous basis compared to more controlled settings (such as in larger camcorders), the resulting video tends to include more non-stationary characteristics, such as higher amounts of motion and variable frame rates. Because of this, accurate frame rate estimation is very difficult if not impossible. The mentioned approaches can also lead to noticeable delays and/or jerky motion in resulting encoded video data.
Thus, ways are desired for improving the processing of video data having variable frame rates in devices, such as mobile devices.